feat: Add Dataset pipelines (experimental)#1933
Conversation
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions! |
| export function getRegisteredDatasetPipelines(): DatasetPipelineDefinition[] { | ||
| return [...registry()]; | ||
| } |
There was a problem hiding this comment.
why do we have this? I think we can remove it, plus it's associated functionality.
There was a problem hiding this comment.
we use this so that you don't have to export each dataset pipeline. it's an analogous thing to how we extract Eval body definitions
There was a problem hiding this comment.
what is the exact use-case? I think I kinda get it but it honestly doesn't feel very JSy
There was a problem hiding this comment.
Oh i see. This is used for bt. Honestly, to avoid API creep (and reduce confusion for SDK users/agents), I would probably just have the global registry and just have bt read off it.
|
Gonna take this over making sure that it's clear to consumers that this is experimental/beta for now. |
Luca Forstner (lforst)
changed the title
Luca Forstner (lforst)
changed the title
Oh great. I landed the python change before seeing this. Do you mind marking it as such in python too, and then whenever you are comfortable, we can land this change as well? |
3 participants
Internal design doc: https://www.notion.so/braintrustdata/Dataset-pipelines-34df78580289803e801eef0ea689bc44